Although a custom coordinate system cannot be published to FME Server, there are two ways they can be made available to the FME Server:
Modify the MyCoordSysDefs.fmw file:
Add its definition to the following file where you have an FME Engine installed:
<FMEServerDir>\Server\fme\Reproject\MyCoordSysDefs.fme
MyCoordSysDefs.fme is a file that contains custom coordinate systems defined by the user.
Add to the shared resource directory:
Place custom coordinate systems in the CoordinateSystems shared resources directory.
Each FME Engine used by FME Server can have a shared resources directory defined for it. The location of the directory is based on the FME_SHARED_RESOURCE_DIR parameter in the Engine's configuration file:
<FMEServerDir>\Server\fmeEngineConfig_<instanceName>.txt
FME_SHARED_RESOURCE_DIR "C:/Apps/FMEServer/Server/Resources/shared"